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Program Description 1 Project Read ® is a multisensory 2 language arts curriculum 

designed for use in a classroom or group setting. Two main 
objectives of the program are to use language in all its forms, 
and to use responsive instruction rather than preplanned 
textbook lessons. The program emphasizes direct instruction, 
and lessons move from letter-sounds to words, sentences, and 
stories. Project Read® has three strands: Phonics/Linguistics, 



Reading Comprehension, and Written Expression, which are 
integrated at all grade levels, though the emphasis of the specific 
strands differs by grade. This intervention report examines the 
effectiveness of Project Read® Phonology. The What Works 
Clearinghouse (WWC) believes that Project Read® Phonology 
was an earlier version of the Phonics/Linguistics strand but was 
unable to confirm this with the developer. 



Research 3 One study (Bussjaeger, 1993) of Project Read® Phonology that 
falls within the scope of the Students with Learning Disabilities 
review protocol meets WWC evidence standards, and one study 
(Acalin, 1995) meets WWC evidence standards with reservations. 
The two studies included 80 students with learning disabilities, 
from kindergarten through grade 5, in six locations. 4 However, 
only the study by Acalin (1995) presents sufficient outcome data 



to allow the WWC to make a determination of the effectiveness 
of Project Read® Phonology. Although this report describes the 
two studies, the WWC review of the effectiveness of Project 
Read® Phonology is based only on the Acalin (1995) study, which 
included 66 students with learning disabilities in kindergarten 
through grade 4 from five school districts. 



1. The descriptive information for this program was obtained from a publicly available source: the program’s website (http://www.projectread.com, down- 
loaded October 2009). The WWC requests developers to review the program description sections for accuracy from their perspective. Further verification 
of the accuracy of the descriptive information for this program is beyond the scope of this review. The intervention was formerly sold as Project Read® 
Phonology for all grades. Project Read® Phonology was used by studies meeting standards both with and without reservations, but it is now sold only as 
either Project Read® Phonics (for grades 1-3) or Project Read® Linguistics (for grades 4-6), both of which fall under Project Read’s Phonology strand. The 
literature search reflects documents publicly available by October 2009. 

2. Project Read ® is one of many curricula that are based, in part, on the principles of the sequential, multisensory Orton-Gillingham approach to teaching 
reading. Other WWC intervention reports related to the multisensory Orton-Gillingham approach include Barton Reading & Spelling System ®, Fundations ®, 
Herman Method™, Orton-Gillingham-based Strategies (Unbranded), Wilson Reading System®, Alphabetic Phonics, and Dyslexia Training Program. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 2.0 (see the WWC Procedures and Standards Handbook, Chapter III), 
as described in protocol Version 2.0. 

4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 
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Research (continued) Based on this one study, the WWC considers the extent of 
evidence for Project Read ® Phonology on students with learn- 
ing disabilities to be small for general reading achievement. 
Bussjaeger (1993) examined outcomes in alphabetics and 
reading comprehension but did not report sufficient outcome 



data to be considered for extent of evidence ratings. No studies 
that meet WWC evidence standards with or without reservations 
examined the effectiveness of Project Read® Phonology on 
students with learning disabilities in the reading fluency, writing, 
math, science, social studies, or progressing in school domains. 



Effectiveness Project Read® Phonology was found to have no discernible effects on general reading achievement for students with learning 
disabilities. Although one of the studies reviewed by the WWC meets evidence standards and collected data in the alphabetics 
and reading comprehension domains, the data were insufficient to confirm the author’s findings in these domains. 



Rating of effectiveness 
Improvement index 5 



General reading 
achievement 


Alphabetics 


Reading comprehension 


No discernible effects 


na 


na 


Average: +5 percentile 


na 


na 


points 




na = not applicable 



Additional program 
information 



Developer and contact 

Developed by Victoria Greene and Mary Lee Enfield in the Bloom- 
ington Public Schools in 1969, Project Read® Phonics/Linguistics 
is distributed by Language Circle Enterprises, Inc. Address: 1620 
West 98th Street, Suite #130, Bloomington, MN 55431. Email: 
languagecircle@projectread.com. Web: www.projectread.com. 
Telephone: (800) 450-0343. Fax: (952) 884-6787. 

Scope of use 

The WWC believes that Project Read® Phonology was an earlier 
version of Project Read® Phonics/Linguistics. According to the 
authors of the two studies that meet WWC standards (Bussjae- 
ger, 1993; Acalin, 1995), Project Read® Phonology is designed 
for students in grades 1-8. According to the developers, Project 
Read® Phonics/Linguistics is designed for students in grades 
1-6 and in special education. 



Teaching 

The program uses concrete examples and multisensory experi- 
ences to teach abstract concepts. It emphasizes strategies such 
as explaining concepts to students before teaching the skill; 
using graphic symbols and body language; and using visual, 
auditory, and tactile exercises to keep students engaged. The 
three program strands (phonics/linguistics, reading comprehen- 
sion, and written expression) are integrated at all grade levels, 
but specific strands are emphasized at certain levels. 

The prekindergarten and kindergarten level focuses on phonics 
and handwriting. Students learn the basics of the ABCs, practicing 
correct pronunciation and tracing the strokes for every letter. The 
primary level (grades 1-3) builds on the phonics foundation and 
introduces reading comprehension strategies, such as identifying 
subjects and key facts. Professional development for teachers and 
educators is also available through Language Circle Enterprises. 



5. These numbers show the average and range of student-level improvement indices for all findings across the studies. 
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Additional program 
information (continued) 



Cost 

Most program components are sold individually. Project Read® 
student workbooks range from approximately $5 to $15; 



classroom materials, from $10 to $15; teacher’s guides and 
assessments, from $35 to $90; and classroom kits, from $295 
to $695. 



Research 



Twenty-five studies reviewed by the WWC investigated the effects 
of Project Read® Phonology on students with learning disabilities. 
One study (Bussjaeger, 1993) is a randomized controlled trial that 
meets WWC evidence standards. One study (Acalin, 1995) used a 
quasi-experimental design that meets WWC evidence standards 
with reservations. The remaining 23 studies do not meet either 
WWC evidence standards or eligibility screens. 

Meets evidence standards 

A randomized controlled trial reviewed by the WWC (Bussjaeger, 
1993) meets WWC evidence standards, but the outcome data 
included in the report from the study are not sufficient 6 to 
allow the WWC to assess the effectiveness of Project Read® 
Phonology. The study included a total of 14 students with learn- 
ing disabilities in grades 4 and 5 from an elementary school in 
southern California. All students in the study were identified as 
learning disabled based on definitions from the Education for All 
Handicapped Children Act and the California Code of Regula- 
tions and had been placed in a special day class by school 
district staff. Students were paired based on matching gender, 
grade level, and pretest reading achievement scores. Then, for 
each pair, one student was assigned to one group and the other 
student to another group. The two groups were then assigned 
randomly either to Project Read® Phonology or to a control 
group. Pretest and posttest data were collected at the start and 
end of the six-week intervention period. All students were from 
low socioeconomic households and were limited English profi- 
cient; six were female and eight were male; six were 4th graders 



and eight were 5th graders. The two groups were instructed in 
the same classroom at the same time, sitting on opposite sides 
of a classroom that was divided by a portable wall. The study 
author and an instructional assistant were trained to deliver 
both interventions and alternated delivery of the interventions 
on a weekly basis. Students with learning disabilities who were 
assigned to Project Read® Phonology received instruction using 
the Project Read Phonology Guide for 20 minutes a day, four 
days a week, for six weeks. Students with learning disabilities in 
the control group participated in “literature-based instruction” for 
20 minutes a day, four days a week, for six weeks. Both groups 
of students participated in regular basal reading programs for 
the remaining 1.5 hours of daily reading instruction. 

Meets evidence standards with reservations 

A second study reviewed by the WWC (Acalin, 1995) used 
a quasi-experimental design. This study included a total of 
66 students with learning disabilities in kindergarten through 
grade 4 from five school districts in three southern California 
counties. Students were identified as learning disabled based 
on a discrepancy between IQ and achievement test scores and 
were placed in one of two programs (Project Read® Phonology 
or Reading Recovery) by school district personnel. Thirty-three 
students with learning disabilities were placed in each group. 
Pretest and posttest data were collected at the start and end of 
a school year. Analysis of pretest scores showed no statistically 
significant or substantively large differences between groups. 

All children in the study were from middle socioeconomic 



6. Bussjaeger (1993) reported raw-score gain scores (posttest mean-pretest mean) for the treatment and comparison groups. Posttest standard deviations 
were reported only for the gain scores, so it was not possible for the WWC to confirm the author’s findings, nor could the WWC calculate an effect size 
or an improvement index. The author also reported posttest grade equivalent means and standard deviations; however, grade equivalent scores are not 
an equal-interval metric, and hence are not appropriate to use in WWC analyses. The author did not respond to a query asking for means and standard 
deviations in a standard score metric. 
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Research (continued) 



Effectiveness 



households, and English was their primary language. The 
sample included 61% male children; 55% Caucasian children, 
36% Hispanic children, and 9% African-American children. 
Students with learning disabilities who participated in Project 
Read ® Phonology received 30 minutes of small-group instruction 
(two to five students) daily for one school year, using the Project 
Read ® Phonology Guide. Students with learning disabilities who 
were assigned to the comparison group participated in Reading 
Recovery 30 minutes daily for the school year, receiving one-on-one 
instruction and using the Rigby Series reading books. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 



Standards Handbook, Appendix G). The extent of evidence takes 
into account the number of studies and the total sample size 
across the studies that meet WWC evidence standards with or 
without reservations. 7 

The WWC considers the extent of evidence for Project Read® 
Phonology for students with learning disabilities to be small 
for general reading achievement. Bussjaeger (1993) examined 
outcomes in alphabetics and reading comprehension but did 
not report sufficient outcome data to be considered for extent 
of evidence ratings. No studies that meet WWC evidence stan- 
dards with or without reservations examined the effectiveness 
of Project Read® Phonology on students with learning disabilities 
in the reading fluency, writing, math, science, social studies, or 
progressing in school domains. 



Findings 

The WWC review of interventions for students with learning 
disabilities addresses student outcomes in nine domains: 
alphabetics, reading fluency, reading comprehension, general 
reading achievement, writing, math, science, social studies, and 
progressing in school. The studies included in this report cover 
three domains: alphabetics, comprehension, and general reading 
achievement. The findings below present the authors’ estimates 
and WWC-calculated estimates of the size and statistical signifi- 
cance of the effects of Project Read® Phonology on students with 
learning disabilities. 8 

Alphabetics. Bussjaeger (1993) assessed the effectiveness of 
Project Read® Phonology on students with learning disabilities 
in the alphabetics domain using the Letter-Word Identification 



subtest from the Woodcock-Johnson Psycho-Educational 
Battery-Revised (WJ-R) and the Reading subtest from the Wide 
Range Achievement Test-Revised (WRAT-R). The study author 
reported no statistically significant effect of Project Read® Pho- 
nology on these measures for students with learning disabilities. 
The WWC was not able to confirm or disconfirm this finding. 

As outcome data were not reported, the WWC was unable to 
determine whether these findings were substantively important. 

Reading comprehension. Bussjaeger (1993) assessed the 
effectiveness of Project Read® Phonology on students with 
learning disabilities in the comprehension domain using the Pas- 
sage Comprehension subtest from the WJ-R. The study author 
reported no statistically significant effect of Project Read® Pho- 
nology on this measure for students with learning disabilities. 9 



7. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for Project Read ® Phonology is in Appendix A5. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the cases of 
Acalin (1995) and Bussjaeger (1993), no corrections for clustering or multiple comparisons were needed. 

9. Bussjaeger (1 993) reported that the intervention group scored better than the comparison group at the p<.1 0 level. 
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Effectiveness (continued) 



The WWC was not able to confirm or disconfirm this finding. 

As outcome data were not reported, the WWC was unable to 
determine whether these findings were substantively important. 

General reading achievement. Acalin (1995) assessed the 
effectiveness of Project Read ® Phonology on students with learn- 
ing disabilities in the general reading achievement domain using 
the Broad Reading cluster (a combination of the Letter-Word 
Identification and Passage Comprehension subtests) from the 
WJ-R. The author’s analysis showed, and the WWC analysis 
confirmed, no statistically significant effect of Project Read ® 
Phonology on this measure for students with learning disabilities. 
The effect size on this measure was not large enough to be 
considered substantively important based on WWC standards. 
According to WWC criteria, this study shows no discernible 



effects of Project Read ® Phonology on students with learning 
disabilities in the general reading achievement domain. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Procedures and Standards Handbook, 
Appendix E). 



The WWC found 
Project Read® Phonology 
to have no discernible 
effects on general reading 
achievement for students 
with learning disabilities. 

No conclusions could 
be reached about 
the effectiveness of 
Project Read® Phonology 
on alphabetics and 
reading comprehension. 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see WWC Procedures 
and Standards Handbook, Appendix F). The improvement index 
represents the difference between the percentile rank of the aver- 
age student in the intervention condition and the percentile rank of 
the average student in the comparison condition. Unlike the rating 
of effectiveness, the improvement index is entirely based on the 
size of the effect, regardless of the statistical significance of the 
effect, the study design, or the analysis. The improvement index 
can take on values between -50 and +50, with positive numbers 
denoting favorable results for the intervention group. 

Based on one study, the improvement index for general read- 
ing achievement is +5 percentile points (using a single measure). 
As Bussjaeger (1993) did not report complete outcome data, the 



improvement index for alphabetics and reading comprehension 
could not be reported. 

Summary 

The WWC reviewed 25 studies on Project Read ® Phonology for 
students with learning disabilities. One of these studies meets 
WWC evidence standards, one study meets WWC evidence 
standards with reservations, and the remaining 23 studies do 
not meet either WWC evidence standards or eligibility screens. 
Based on the one study that reported complete outcome data, 
the WWC found no discernible effects on general reading 
achievement for students with learning disabilities. Although a 
second study reviewed by the WWC meets evidence standards 
and collected data in the alphabetics and reading comprehen- 
sion domains, no data were available to confirm the author’s 
findings in these domains. The conclusions presented in this 
report may change as new research emerges. 
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Appendix A1.1 Study characteristics: Bussjaeger, 1993 



Characteristic 


Description 


Study citation 


Bussjaeger, J. J. (1993). The effectiveness of Project Read on the reading achievement of students with learning disabilities (Master’s thesis, California State University, Ful- 
lerton, 1993). Masters Abstracts International, 31 { 04), 54-1480. 


Participants 


The sample for this study included a total of 14 students with learning disabilities in grades 4 and 5. All students in the study were identified as learning disabled based on defini- 
tions from the Education for All Handicapped Children Act and the California Code of Regulations and had been placed in a special day class by school district staff. Two groups 
of students were formed by matching pairs on gender, grade level, and pretest reading achievement scores. The two groups were then assigned randomly to intervention (Project 
Read® Phonology] or control (literature-based) conditions. Prior to the study, all of the participating students received one month of instruction in Project Read® Phonology and at 
least one year of literature-based instruction. Pretest and posttest data were collected at the start and end of the six-week intervention period by the study author. All students were 
from low socioeconomic households and were limited English proficient; six were female and eight were male; six were 4th graders and eight were 5th graders. 


Setting 


The study was conducted with students with learning disabilities from one elementary school in southern California. Students were enrolled in grades 4 and 5. The two study 
groups were instructed in the same classroom, at the same time, with the two groups sitting at opposite sides of the classroom, which was divided by a portable wall. 


Intervention 


Students with learning disabilities who were assigned to Project Read® Phonology received instruction using the Project Read® Phonology Guide. Project Read® Phonology 
instruction was delivered 20 minutes a day, four days a week, for six weeks. Project Read® Phonology students participated in regular basal reading programs for the remain- 
ing 1.5 hours of daily reading instruction. 


Comparison 


Students with learning disabilities in the comparison group participated in “literature-based instruction” for 20 minutes a day, four days a week, for six weeks. Comparison 
students also participated in regular basal reading programs for the remaining 1.5 hours of daily reading instruction. 


Primary outcomes 
and measurement 


The primary outcome domains assessed were alphabetics and reading comprehension. Alphabetics (letter knowledge) was measured by administration of the Letter-Word 
Identification subtest of the Woodcock-Johnson Psycho-Educational Battery-Revised (WJ-R) and the Reading subtest of the Wide Range Achievement Test-Revised 
(WRAT-R). Reading comprehension was measured by administration of the Passage Comprehension subtest of the WJ-R. The assessments were individually administered 
in English. Pretesting and posttesting were done prior to and immediately following the six-week intervention period. For each of these outcomes, the author reported gain 
scores (posttest mean-pretest mean) for the Project Read® Phonology and comparison groups. Posttest standard deviations were not reported, so it was not possible for the 
WWC to confirm the author’s findings, nor could the WWC calculate an effect size or an improvement index. For a more detailed description of these outcome measures, see 
Appendices A2.1-A2. 2. 


Staff/teacher training 


The study author was the lead teacher and received 12 hours of training in the phonology component of Project Read®, plus another 18 hours of training in the reading 
comprehension and written expression components of Project Read® . An instructional assistant was trained in the use of Project Read® Phonology by the lead teacher for six 
hours. The lead teacher and instructional assistant also participated in 40 hours of workshop training in literature-based and whole language instruction. The lead teacher and 
instructional assistant alternated weekly between the Project Read® Phonology group and the literature-based comparison group. The lead teacher was a credentialed special 
education teacher. Both instructors had 10 years of experience working with students with learning disabilities. 
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Appendix A1.2 Study characteristics: Acalin, 1995 



Characteristic 


Description 


Study citation 


Acalin, T. A. (1995). A comparison of Reading Recovery to Project Read (Master’s thesis, California State University, Fullerton, 1995). Masters Abstracts International, 33(06), 
54-1660. 


Participants 


The sample for this study included a total of 66 students with learning disabilities in kindergarten through grade 4. Students were identified as learning disabled based on defini- 
tions from the Education for All Handicapped Children Act and the California Code of Regulations and were placed in one of two programs (Project Read® Phonology or Reading 
Recovery) by school district personnel. Thirty-three students with learning disabilities were placed in each group. Pretest and posttest data were collected at the start and end of 
a school year by Resource Specialists. For analysis purposes, pairs of students were formed by matching on gender, grade level, ethnicity, and pretest score. Analysis of pretest 
scores showed no statistically significant or substantively large differences between groups. All children in the study were from middle socioeconomic households, and English was 
their primary language. The sample included 61% male children; 55% Caucasian children, 36% Hispanic children, and 9% African-American children. 


Setting 


The study was conducted with students with learning disabilities from five school districts in three southern California counties — Orange, San Bernardino, and Los Angeles. 


Intervention 


Students with learning disabilities who participated in Project Read® Phonology received 30 minutes of small-group instruction (two to five students) daily for one school year, 
using the Project Read® Phonology Guide. All instruction was conducted by credentialed Resource Specialists who had five or more years of experience working with students 
with learning disabilities. Teachers followed the Project Read® Phonology manuals, lesson by lesson, with minimal program adaptations. 


Comparison 


Students with learning disabilities who were in the comparison group participated in Reading Recovery. In this study, students with learning disabilities participated in Reading 
Recovery 30 minutes daily, receiving one-on-one instruction and using the Rigby Series reading books. 


Primary outcomes 
and measurement 


The primary outcome domain assessed was general reading achievement, which was measured by combining the scores of two subtests from the Woodcock- Johnson 
Psycho-Educational Battery-Revised: the Letter-Word Identification subtest and the Passage Comprehension subtest. Together, these subtests form the Broad Reading cluster 
of the Woodcock-Johnson battery. The assessment was administered in English by Resource Specialists. Pretesting was done in the fall of the school year and posttesting was 
done in the spring. For a more detailed description of this outcome measure, see Appendix A2.3. 


Staff/teacher training 


Project Read® Phonology teachers received the full training associated with this program (three inservice days — one for phonology, one for comprehension, and one for 
written language). 
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Appendix A2.1 Outcome measures for the alphabetics domain 



Outcome measure Description 

Letter-Word Identification The WJ-R Letter-Word Identification subtest measures a student’s skill in identifying individual letters and words. The examiner shows the student letters or words and the 
subtest of the Woodcock- student responds with the letter or name of the word (as cited in Bussjaeger, 1993) (http://www.riverpub.com/products/wjlllComplete/). 

Johnson Psycho-Educational 
Battery-Revised (WJ-R) 



Reading subtest of the 
Wide Range Achievement 
Test-Revised (WRAT-R) 


The WRAT-R Reading subtest measures skills in letter recognition, letter naming, and pronunciation of words in isolation. The test consists of 15 letters and 42 words 
that the student is asked to name or pronounce (as cited in Bussjaeger, 1993) (http://portal.wpspublish.com/portal/page?_pageid=53,118660&_dad=portal&_ 
schema=PORTAL&cmp=20_google&kw=wide%20range%20achievement%20test&gclid=CIXA5_eEn54CFQnxDAodAWTQIw). 


Appendix A2.2 Outcome measures for the reading comprehension domain 


Outcome measure 


Description 


Passage Comprehension 
subtest of the WJ-R 


The WJ-R Passage Comprehension subtest measures a student’s ability to identify missing keywords in a reading passage. The task requires the student to state a word that 
would be appropriate in the context of the passage (as cited in Bussjaeger, 1993) (http://www.riverpub.com/products/wjlllComplete/). 


Appendix A2.3 Outcome measures for the general reading achievement domain 


Outcome measure 


Description 


Broad Reading cluster 
from the WJ-R 


The WJ-R Broad Reading cluster is a combination of the Letter-Word Identification and Passage Comprehension subtests and provides a broad measure of reading achieve- 
ment (as cited in Acalin, 1995) (http://www.riverpub.com/products/wjlllComplete/). 



WWC Intervention Report Project Read® Phonology 



July 2010 



10 







Appendix A3.1 Summary of study findings included in the rating for the alphabetics domain 1 









Author’s findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Project Read® 

Phonology Comparison 

group 3 group 


Mean 

difference 4 
(Project Read® 
Phonology - 
comparison; 


Statistical 

Effect significance 6 

size 5 (at a = 0.05) 


Improvement 

index 7 



WJ-R Letter-Word Identification 
subtest 


Grades 
4 and 5 


14 


3.57 

(nr) 


Bussjaeger, 1993 8 

1.57 

(nr) 


2.00 


na 


na 


na 


WRAT-R Reading subtest 


Grades 
4 and 5 


14 


3.71 

(nr) 


1.43 

(nr) 


2.28 


na 


na 


na 


Domain average for alphabetics 9 












na 


na 


na 



na = not applicable 

nr = not reported 

WJ-R = Woodcock-Johnson Psycho-Educational Battery-Revised 

WRAT-R = Wide Range Achievement Test-Revised 

1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the alphabetics domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. Posttest standard deviations were not reported for this study, and the author did not respond to a query asking for means and standard deviations in a standard 
score metric. 

3. Bussjaeger (1993) reported gain scores (posttest mean-pretest mean) for the treatment and comparison groups. Standard deviations for the gain scores are reported, but student-level posttest 
standard deviations are needed to calculate an effect size and improvement index. These were not available for the present study. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. Posttest standard deviations were not reported by the author, so it was not pos- 
sible for the WWC to calculate an effect size or an improvement index. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. Statistical significance was calculated by the 
author using gain score standard deviations and showed that students in the Project Read® Phonology group gained significantly more (p<.05) than students in the comparison group on the 
WJ-R Letter-Word Identification subtest, but not on the WRAT-R Reading subtest. Posttest standard deviations were not reported by the author, so it was not possible for the WWC to compute 
statistical significance in order to confirm or disconfirm the study findings. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Bussjaeger (1993), no corrections for clustering or multiple comparisons were needed. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A3.2 Summary of study findings included in the rating for the reading comprehension domain 1 









Author’s findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Project Read® 

Phonology Comparison 

group 3 group 


Mean 

difference 4 
(Project Read® 
Phonology - 
comparison; 


Statistical 

Effect significance 6 

size 5 (at a = 0.05) 


Improvement 

index 7 



Bussjaeger, 1993 8 



WJ-R Passage 


Grades 


14 


0.71 


0.42 


0.29 


na 


na 


na 


Comprehension subtest 


4 and 5 




(nr) 


(nr) 










Domain average for reading comprehension 9 










na 


na 


na 



na = not applicable 
nr = not reported 

WJ-R = Woodcock-Johnson Psycho-Educational Battery-Revised 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the reading comprehension domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. Posttest standard deviations were not reported for this study. 

3. Bussjaeger (1993) reported gain scores (posttest mean-pretest mean) for the treatment and comparison groups. Standard deviations for the gain scores are reported, but student-level posttest 
standard deviations are needed to calculate an effect size and improvement index. These were not available for the present study. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. Posttest standard deviations were not reported by the author, so it was not pos- 
sible for the WWC to calculate an effect size or an improvement index. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. Statistical significance was calculated by the 
author using gain score standard deviations and showed that students in the Project Read ® Phonology group did not have statistically significant gains compared to students in the comparison 
group on the WJ-R Passage Comprehension subtest. Posttest standard deviations were not reported by the author, so it was not possible for the WWC to compute statistical significance in 
order to confirm or disconfirm the study findings. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Bussjaeger (1993), no corrections for clustering or multiple comparisons were needed. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A3.3 Summary of study findings included in the rating for the general reading achievement domain 1 









Author’s findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Project Read® 

Phonology Comparison 

group 3 group 


Mean 

difference 4 
(Project Read® 
Phonology - 
comparison; 


Statistical 

Effect significance 6 

size 5 (at a = 0.05) 


Improvement 

index 7 



Acalin, 1995 8 



WJ-R Broad Reading cluster K through grade 4 


66 


82.91 

(11.00) 


81.54 

(11.94) 


1.37 


0.12 


ns 


+5 


Domain average for general reading achievement 9 










0.12 


na 


+5 



ns = not statistically significant 
na = not applicable 

WJ-R = Woodcock-Johnson Psycho-Educational Battery-Revised 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the general reading achievement domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 

had more similar outcomes. Posttest standard deviations were not reported by Acalin (1995); however, the posttest scores for each student in the study were reported. The WWC used the 
student-level posttest scores to calculate posttest standard deviations for the Project Read ® and comparison groups. 

3. For Acalin (1995), each intervention group mean is calculated as the unadjusted control mean plus the adjusted mean difference as calculated by the WWC. Standard deviations are unadjusted 
and were calculated by the WWC based on raw data contained in the report. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 

The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Acalin (1995), no corrections for clustering or multiple comparisons were needed. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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